Statistical Mechanics of Learning in a Large Committee Machine
نویسندگان
چکیده
In a We use statistical mechanics to study generalization in large committee machines. For an architecture with nonoverlapping receptive fields a replica calculation yields the generalization error in the limit of a large number of hidden units. For continuous weights the generalization error falls off asymptotically inversely proportional to Q, the number of training examples per weight. For binary weights we find a discontinuous transition from poor to perfect generalization followed by a wide region of metastability. Broken replica symmetry is found within this region at low temperatures. For a fully connected architecture the generalization error is calculated within the annealed approximation. For both binary and continuous weights we find transitions from a symmetric state to one with specialized hidden units, accompanied by discontinuous drops in the generalization error.
منابع مشابه
Query learning for maximum information gain in amulti - layer neural
In supervised learning, the redundancy contained in random examples can be avoided by learning from queries, where training examples are chosen to be maximally informative. Using the tools of statistical mechanics, we analyse query learning in a simple multi-layer network, namely, a large tree-committee machine. The generalization error is found to decrease exponentially with the number of trai...
متن کاملLETTER TO THE EDITOR Globally optimal on-line learning rules for multi-layer neural networks
We present a method for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework. This rule maximizes the total reduction in generalization error over the whole learning process. A simple example demonstrates that the locally optimal rule, which maximizes the rate of decrease in generalization error, may perform poorly in compar...
متن کاملGlobally Optimal On-line Learning Rules
We present a method for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework. This work complements previous results on locally optimal rules, where only the rate of change in generalization error was considered. We maximize the total reduction in generalization error over the whole learning process and show how the resultin...
متن کاملDiscontinuous Generalization in Large Committee Machines
J. Hertz Nordita Blegdamsvej 17 2100 Copenhagen 0 Denmark The problem of learning from examples in multilayer networks is studied within the framework of statistical mechanics. Using the replica formalism we calculate the average generalization error of a fully connected committee machine in the limit of a large number of hidden units. If the number of training examples is proportional to the n...
متن کاملJ un 1 99 5 Unrealizable Learning in Binary Feed - Forward Neural Networks DRAFT February 1 , 2008
Statistical mechanics is used to study unrealizable generalization in two large feed-forward neural networks with binary weights and output, a perceptron and a tree committee machine. The student is trained by a teacher being larger, i.e. having more units than the student. It is shown that this is the same as using training data corrupted by Gaussian noise. Each machine is considered in the hi...
متن کامل